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WO 94/24631 PCT/US94/02572 

COMPUTER GRAPHIC AND LIVE VIDEO SYSTEM FOR ENHANCING 
VISUALIZATION OF BODY STRUCTURES DURING SURGERY 



CROSS REFERENCE TO RELA TED APPLICATIONS 

This application is related to applications "System For Displaying 
Solid Cuts For Surfaces of Solid Models" by William E. lorensen, 
Harvey E. Cline, Bruce Teeter, and Siegwalt Ludke, Serial No. 
07/812,394, "Solid Models Generation By Span Method Using Dividing 
5 Cubes," by Harvey E. Cline, William E. Lorensen and Siegwalt Ludke, 
Serial No. 07/812,479, and "Apparatus and Method For Displaying 
Surgical Cuts in Three-Dimensional Models" by Harvey E. Cline, William 
E. Lorensen and Siegwalt Ludke, Serial No. 07/812,395, all filed 
December 23, 1991, and all assigned to the present assignee. 
10 BACKGROUND OF THE INVENTION 

1. Held of the Invention 

The present invention relates to a system for aiding a 
surgeon in visualizing body structures during surgery and more 
specifically to a system which displays, in real time, interactive images 
15 of both internal and external body structures to aid in surgery. 

2. Discussion of Prior Art 

Presently, during surgery, surgeons view several static 
views of the patient in the operating room. Typically these are 
transparencies of magnetic resonance (MR), computer tomography 

20 (CT), or ultrasound images. Since these images are two dimensional 
static images, the surgeons must determine actual three-dimensional 
(3D) location and shape of desired internal structures within the patient 
from the 2D images which they are viewing. Conceptually, the 
surgeon constructs a 3D model of the internal structures and correlates 

25 these internal structures with visible external structures of the patient 
where they must cut. This is often difficult because the scale and the 
orientation of the 2D image may differ from what the surgeon is 
seeing, and the surgeon may not be able to view both the 
patient and the medical diagnostic images 
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simultaneously . 

Another technique employed in localization of internal 
structures during surgery is known as stereotactic surgery as 
described in "Interactive Stereotactic Surgical System for the 

5 Removal of Intracranial Tumors Utilizing the CO2 Laser and CT- 
Derived Database" by B. A. Kail, P. J. Kelly, and S. J. Goerss, IEEE 
Transactions on Biomedical Engineering, vol. BME-32, no. 2, pp 112- 
116, 1985; and "Comprehensive Computer- Assisted Data Collection 
Treatment Planning and Interactive Surgery" by B. A, Kail, P. J 

10 Kelly, and S* J. Goerss, Medical Imaging, vol. 767 pp. 509-514, 1987. 
With this approach, a rigid mechanical frame is attached to the 
patient before a CT or MR procedure. The frame and its landmarks 
can be seen in the resulting images. Mechanisms on the frame 
position a probe at specific location within the image. The 

15 disadvantages of this approach are that the frame limits access to the 
patient, and the images are static images which do not follow the 
patient if he moves during surgery. 

A third technique used for localization of internal structures is 
described in "A Frameless Stereotaxic Operating Microscope for 
20 Neurosurgery" by E. M. Friets, J. W. Strohbehn, J. F. Hatch, and D. 
W. Roberts, IEEE Transactions on Biomedical Engineering, vol. 36., 
no. 6, pp 608-617, June 1989. 

Three dimensional models of anatomy can be created from 
data of different medical imaging modalities as described in the 
25 application listed above in the "CROSS REFERENCE TO RELATED 
APPLICATIONS". These applications describe creating and 
manipulating models of internal structures of patients and providing 
images of selected structures at desired orientations to an operator. 
These allow visualization of internal structures as solid models. 

30 Currently there is a need for a system to aid surgeons in 

surgery which interactively displays computer generated models of 
internal structures in correct relation with external structures of the 
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patient to guide during surgery. 

OBJECTS OF THE TNVENTTfN 

An object of the present invention is to provide a system which 
aides in surgery by interactively displaying an image of external 
5 structures superimposed upon internal structures, even if the 
patient changes his position. 

Another object of the present invention is to provide an 
interactive system which displays desired external and internal 
structure, both having the same scale and viewed from the same 
10 orientation angles. 

Another object of the present invention is to provide a guide 
image of normal anatomy superimposed on body structures of a 
patient to aid surgeons in performing reconstructive surgery. 



STMttRY OF THE 



15 A real-time surgery apparatus for displaying interactive 

internal and external images of a patient employs a video camera for 
providing real-time images of exposed surfaces of the patient. 

A medical imaging device obtains three-dimensional (3D) 
imaging data of internal structures of said patient which are fed to a 
20 workstation. The workstation creates three-dimensional (3D) 
computer generated models which may be manipulated without 
further need for the medical imaging device. Computer generated 
images of the models are interactively oriented and scaled to coincide 
with the real-time video images. 

25 A video mixer mixes portions of the computer generated image 

and the live video to display external structures of the patient 
superimposed over the computer generated images of internal 
structures in their correct relationship. 
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BRIEF DFSCRTPTTCN CP THE DRAWTTO^q 

The features of the invention believed to be novel are set forth 
with particularity in the appended claims. The invention itself, 
however, both as to organization and method of operation, together 
5 with further objects and advantages thereof, may best be understood 
by reference to the following description taken in conjunction with 
the accompanying drawing in which: 

Fig. 1 is a simplified block diagram of a first embodiment of a 
surgical apparatus according to the present invention. 

10 Fig. 2 is a simplified block diagram of a second embodiment of a 
surgical apparatus according to the present invention. 

Fig. 3 is a simplified block diagram of the workstation of the surgical 
apparatus of Figs. 1 and 2. 

Figure 4 shows a computer generated image of 3D model of a patient 
15 created from MR slice data. 

Figure 5 is a frame of the live video from the video camera used to 
monitor a patient. 

Figures 6 and 7 each illustrate a frame of the interactive image video 
wherein half of the image is a computer generated image, and the 
20 other half is a display from the video cameras. 

Figure 8 is a frame of the interactive video illustrating the surgeon's 
tracing of important sulci of the patient's brain on the patient's 
skull. 

Figure 9 is a frame of the interactive video illustrating the surgeon 
25 marking of a surgical entry point which would allow access to a 
desired sulcus on the patient's skull. 

Figure 10 is a frame from the video cameras showing the markings 
on the patients skull of the important sulci and entry point drawn 
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in the example. 

\ 

DETAILED nKSCRTPTTfM C P THE TNVRMTTnM . 

In Pig. 1, a patient 1 on which surgery is to be performed, is 
scanned by a medical imaging apparatus such as a magnetic 
5 resonance (MR) imaging apparatus, a computed axial tomography 
(CAT) apparatus, a positron emission tomography (PET) or similar 
imaging device capable of creating volumetric data, or 3-dimensional 
(3-D) data, from internal structures of the patient. After imaging, 
apparatus 10 provides the volumetric data to a model workstation 

10 100. Once the volumetric data has been provided to model 

workstation 100, there is no further need for imaging apparatus 10. 
This is important since it does not require surgery to then be planned 
and performed with the patient situated within the confines of an 
imaging apparatus, which is very constricting in the case of MR 

15 imaging. Model workstation 100 stores the volumetric data and 
creates computer generated models from the data capable of being 
scaled, rotated and otherwise manipulated, without the further need 
for imaging apparatus 10. 

An imaging device capable of sensing external structures, 
20 such as a live video camera 12, monitors patient 1. A tracking device 
50 coupled to both video camera 12 and patient 1 determines a relative 
roll a, pitch 9, and yaw $ orientation between video camera 12 and 
subject L Tracking device 50 may be a 6-degrees of freedom tracking 
device as described in "The Flock of Birds" Installation and 
25 Operation Guide, Ascension Technology Corporation, July 5, 1992. 
Tracking device 50 also determines a location (in Cartesian 
coordinates) of video camera 12 with relation to patient 1. Patient 1 is 
assumed to be at the origin of the Cartesian coordinate system (x,y,z) 
= (0,0,0), therefore all distances relative to the patient are simply the 
30 (x,y,z) location. The location and orientation are interactively 

provided to model workstation 100 by tracking device 50. The location 
and orientation may also be provided manually to model 
workstation(s) in different embodiments. 
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Model workstation 100 processes the three-dimensional (3D) 
volumetric data it receives and determines surfaces between 
differing types of tissue. Connectivity of similar types of tissue 
adjacent to one another is then determined. This is known as 
5 segmentation. When the 3-D volumetric data has been segmented 
into internal structures, each internal structure may be treated as a 
separate solid object by model workstation 100. The model 
workstation has the capability of selectively displaying desired 
internal structures, color coding structures and severing, rotating 
10 and translating internal structures in order to manipulate the 
internal structures in a desired manner to provide visualization to 
an operator working model workstation 100. 

Model workstation 100 receives input data from a model cut 
plane input device 40 and a workstation view input device 60 to select 

15 the method of displaying internal structures of patient 1. Tracking 
device 50 provides relative orientation data between video camera 12 
and patient 1 and scaling, which allows model workstation 100 to 
synthesize an interactive computer generated image of internal 
structures of patient 1 and have it coincide with the real-time image 

20 from video camera 12. This interactive computer generated image is 
converted from a computer monitor signal, such as an RGB 
computer monitor signal, to a video format as it is passed through a 
scan converter 192. The computer generated image, in video format, 
is provided to a video mixer 194. A real-time video signal from video 

25 camera 12 is also passed to video mixer 194. Video mixer 194 

provides a desired mixture of the video signal from the video camera 
12 with the computer generated image originally from model 
workstation 100. Video mixer 194 may receive input signals from 
the operator, for example, through a workstation view input 60. This 

30 may involve the degree of transparency of each image, or any of 
various other special effects. One very useful special effect is a 
movable window which has 0% transparency (100% opaqueness) 
superimposed upon another image. When the window image is of 
internal structures superimposed upon external structures, it 
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creates the illusion of external structures cut away within the 
window, exposing underlying internal structures. Other video 
special effects such as a "50/50 mix" and a "side to side wipe" 
described in ,r Video Toaster System 2.0 Software Manual" 1992 by J. 

5 M. Herbert, NewTek Inc. may also be employed. Video mixer 194 
provides real-time video signal from video camera 12 and the 
computer generated image of internal structures to a monitor 250, 
allowing both internal and external structures to be visible 
simultaneously. The resulting image displayed on monitor 250 

10 provides an interactive real-time image of patient, even if the patient 
moves during surgery. Since internal structures and their relation 
to exposed external structures are simultaneously displayed, the 
surgeon perceives a very accurate indication of where he should cut 
through external structures to arrive at a desired internal structure 

15 while avoiding vital internal structures. 

Figure 2 illustrates an alternative embodiment of the present 
invention in which a pair of video cameras 12a, 12b create a real-time 
video image of external structures of patient 1. Video cameras 12a, 
12b differ in their orientation with relation to the patient to simulate a 

20 stereoscopic view with video camera 12a corresponding to a right 
view from an operator's right eye, and video camera 12b 
corresponding to a left view from an operator's left eye. Video 
cameras 12a and 12b together provide a stereoscopic view of external 
structures of patient 1. Tracking device 50 also tracks the relative 

25 location (xi,yi,zi) and orientation angle (at, $i, 6i) between video 
camera 12a and patient 1. Tracking device 50 tracks a second 
location (x2,y2»Z2) and orientation (<X2,4>2»02) between video camera 12b 
and patient 1. The locations and orientations are fed to model 
workstation 100a 9 100b which create a right and left computer 

30 graphic image at locations (xi,yi t zi), (X2,y2,z2), respectively, and 
orientations (oti, <j>i, 8i) (ot2><t>2»02)t respectively, corresponding to the 
views of video cameras 12a, 12b f respectively. 

Video cameras 12a and 12b each pass a real-time video signal 
to a sequencer 196 which selects a signal from one of the two video 
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cameras and passes it to mixer 194, then selects the signal from the 
other camera and passes it to mixer 194* Sequencer 196 alternates 
between the video signal from video cameras 12a, 12b in a regular 
fashion several times per second, for example 30 times per second, so 
5 that each video signal is provided at 1/2 the video frame rate. 

Model workstations 100a, 100b pass a left and right computer 
generated image, respectively, pertaining to a left and right view, 
respectively, to scan converters 192a, 192b, respectively which convert 
the computer generated image signals to video format, and pass the 

10 converted computer generated signals to a sequencer 198. 

Sequencers 196 and 198 may be conventional video sequencers as 
described in "Portable, Low Cost Devices for Videotaping, Editing 
and Displaying Field Sequential Stereoscopic Motion Pictures and 
Video" by M. Starks, Stereoscopic Displays and Applications Proc. 

15 SPIE Vol. 1256, pp. 266-271, 1990. 

Sequencer 198 passes the left computer generated image to 
video mixer 194 at the same time sequencer 196 passes the video 
signal corresponding to the left view to video mixer 194. Sequencer 
196 then passes the video image from camera 12a corresponding to 
20 the right view to mixer 194 at the same time that sequencer 198 

passes the right computer generated image to mixer 194. Sequencers 
196 and 198 alternate many times per second, in synchronization, 
between right and left views. Video mixer 194 mixes portions of the 
signals it receives and displays them on a monitor 250. 

25 The image on monitor 250 is time multiplexed to produce an 

image to the left eye and right eye of the operator in an alternating 
fashion. A stereoscopic viewer 252 is synchronized to a sync pickup 
254 which monitors a video sync signal sent to monitor 250 and 
operates to block the vision of the operator's left or right eye allowing 

30 the opposite eye to view the image on screen 250 for an instant and 
vice-versa. This allows the operator to see the left image with the left 
eye while the right eye sees nothing and the right image with the 
right eye while the left eye sees nothing in rapid succession. This 
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creates a stereoscopic illusion, adding the dimension of depth 
perception in viewing the image displayed on monitor 250. Depth 
perception is very valuable in surgery since it adds a dimension that 
assists in visually localizing structures, which is especially 
5 important in complex, delicate surgery. 

Once both the real-time video and the computer generated 
image(s) are visible, the computer image is initialized. Initialization 
may be accomplished by manual input from the operator to rotate, 
translate and scale the computer generated image(s) until they 
10 match the live video image(s), or by employing tracking device 50 to 
set initial parameters. 

Once the 3D model and the live video are aligned, tracking 
device 50 keeps the view angles and field of view consistent. This 
allows real-time interactive synchronization between the video 
15 image(&) and the computer generated image(s). 

Figure 3 is a more detailed block diagram of model workstation 
100 and 100a, 100b of Figures 1 and 2, respectively. 3-D volumetric 
data acquired from imaging apparatus 10 of Figs. 1 and 2 is passed to 
a segmentation processor 32 which may employ a number of 

20 methods of differentiating tissue types and internal structures from 
the volumetric data. U.S. Patent 5,187,658 issued February 16, 1993 
"System and Method for Segmenting Internal Structures Contained 
Within the Interior Region of the Solid Object" by Harvey E.Cline and 
William E. Lorensen, assigned to the present assignee and hereby 

25 incorporated by reference, describes a method compatible with the 
present invention for categorizing tissue into internal structures 
from 3-D volumetric data obtained from a medical imaging device. 
U.S. Patent 07/631,121 filed 12/20/90 "Segmentation of Stationary and 
Vascular Surfaces in Magnetic Resonant Imaging" by Harvey E. 

30 Cline , Steven P. Souza and William E. Lorensen also assigned to the 
present assignee and hereby incorporated by reference describes 
segmenting not only stationary tissue into internal structures, but 
also segmenting flowing fluids into vascular structures. 
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Segmentation processor 32 may employ a memory 30 as a storage 
and scratch pad area in determining segmentation of internal 
structures. 

Once the internal structures have been segmented, 
5 segmentation processor 32 places the image in the proper form for 
display by a custom model manipulation circuit 5. Segmentation 
processor 32 may employ the "dividing cubes" method described in 
U.S. Patent 4,719,585 issued January 12, 1988 'Dividing Cubes 
System and Method for the Display of Surface Structures Contained 
10 with the Interior Region of a Solid Body, by Harvey E. Cline, Siegwalt 
Ludke and William E. Lorensen assigned to present assignee and 
incorporated by reference. The dividing cubes method creates point 
and normal vectors to describe the surfaces between internal 
structures. 

15 The internal structures must then be displayed in near real- 

time to generate a computer generated image at a proper size, 
location and orientation by model manipulation circuit 5. Tracking 
device 50 (Figs. 1 and 2) provides to a model manipulation circuit 5 
the location and orientation of video camera 12 in the embodiment of 

20 Fig 1, and video cameras 12a, 12b in Fig. 2 with respect to patient 1. 
Model manipulation circuit 5 then creates and provides a computer 
generated image, or images in the case of Fig. 2, to the scan 
converter 192 (Figs. 1 and 2). 

Model manipulation circuit 5 may comprise the accelerator 
25 board described in U.S. Patent 4,985,834 "System and Method 

Employing Pipelined Parallel Circuit Architecture for Displaying 
Surface Structures of the Interior Region of a Solid Body" by Harvey 
E. Cline, Richard I. Hartley, Siegwalt Ludke and Sharbel E. Noujaim 
issued January 15, 1991 assigned to the present assignee and hereby 
30 incorporated by reference. 

Model manipulation circuit 5 of Fig. 3 may alternatively 
comprise the accelerator circuit described in U.S. Patent 
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Application, Ser. No. 07/812,479 filed 12/23/91 "Solid Model 
Generation by Span Method Using Dividing Cubes" by Harvey E. 
Cline f William E. Lorensen and Siewalt Ludke, assigned to the 
present assignee and hereby incorporated by reference. 

5 Since each internal structure is segmented into what resemble 

solid objects, they may be manipulated as solid objects. In the case of 
structures of a patient, a surgeon may acquire data from the patient 
by medical imaging, then plan surgery by manipulating the models 
to plan a desired result before surgery. This is common in complex 
10 reconstructive surgery. Once the plan is determined, it may be 

stored and played back during surgery. The internal structures are 
interactively oriented and scaled to coincide with the actual patient. 

A user employs as a model cut plane input device, a 
workstation view input device (40, 60 of Pigs. 1 and 2, respectively) to 

15 select planes in which to cut the structures in the model, to select a 
three-dimensional orientation of the model, and to select screen cut 
planes which define a workstation viewing region. A model clipping 
circuit 70 determines points within the model cut planes. A rotate 
circuit 130 rotates point and normal vectors received from 

20 segmentation processor 32 and determines shading based upon the 
orientation of the normal vector at each point. A screen clipping 
circuit 150 determines points within a region defined by screen cut 
planes. A display circuit 7 displays a three-dimensional image of 
multiple surfaces that are within the desired display region and the 

25 screen cut planes. 

In radical surgery such as ablative surgery, or massive 
trauma cases, there is little structure which remains to correctly 
determine what a normal anatomy should be. In these cases, an 
additional model workstation may have a model of normal structures 
30 stored which may be mixed with the other images being displayed to 
act as a guide in reconstructive surgery. This may be implemented 
by additional workstations or model manipulation boards. 
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Display circuit 7 displays exposed surfaces of an image created 
from the transformed points within the desired region. The image 
resembles a solid object with superficial surfaces being visible and 
surfaces which lie behind the superficial surfaces being hidden. The 
5 display circuit is comprised of depth buffer memory 180, AND circuit 
220, image buffer memory 190 and a compare circuit 210. 
Compare circuit 210 receives a depth coordinate £ along a bus 200 and 
compares the depth £ with a present value in the depth buffer 180 at 
address If the depth £ of the present point * 8 closer to the 

10 viewer (i.e., has a smaller value) than the previous depth stored in 
depth buffer memory 180 at address then compare circuit 210 
sets a depth line 202 connected to AND circuit 220 'high'. 

AND circuit 220 sets an 'update' line 225 connected to both the 
depth buffer memory 180 and image buffer memory 190 'high' when 
15 it senses lines 80, 160 and 202 all being Tugh\ causing the pixel 

intensity pix to be stored at address of the image buffer memory, 
thereby shading an image displayed on monitor 250 (Figs. 1 and 2). 
This also causes depth C to be stored at address (l;A|f) of depth buffer 
memory, thereby updating the present depth at address 

20 The workstation is capable of displaying the models as shaded 

three-dimensional solid surfaces redrawn at a rate of several times 
per second on a low power graphics workstation, allowing near- 
realtime manipulation. 

Redaction to Praotf re 

25 The data transformations, computer graphics, user interface, 

and stereo portions of this invention were implemented with General 
Electric Co.'s object-oriented software development system, LYMB. 
An embodiment of the present invention was constructed employing 
a Silicon Graphics Reality Engine workstation and a SUN 

30 Sparcstation with the custom accelerator board employing the 

dividing cubes technique. The video portions of this invention were 
implemented using a RGB videolink NTSC scan converter, Parallax 
video board, Newtek Video toaster, 3DTV field sequencer, 
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commercial video cameras and VCR, 3DTV Stereo glasses and 
driver. 

To demonstrate surgical planning applied to brain surgery, 
the Central sulcus, Postcentral sulcus and the Superior temporal 
5 sulcus of a patient were located with the aid of the present invention. 

The first step of the technique is to align live video of the patient 
with a 3D model of the patient to initialize the system. Figure 4 
shows a computer generated image of a 3D model of a patient created 
from MR slice data. Figure 5 is a frame of the live video from the 
10 video camera monitoring the patient. Figures 6 and 7 illustrate a 
"side to side wipe" frame of the interactive image video wherein half 
of the image is a computer generated image, and the other half is 
derived from the video cameras, allowing the patient to be aligned 
with the 3D MR model. 

15 The second step of our technique is to superimpose semi- 

transparent live video image(s) over semi-transparent computer 
generated imaged), allowing the surgeon to see the external 
structures of the patient, a 3D model of the internal structures of the 
patient, and the surgeon's hands and instruments. Figure 8 

20 illustrates the surgeon's tracing of important sulci of the patient's 
brain on the patient's skull (in this case, simulated by a swimming 
cap). Figure 9 illustrates the surgeon's marking on the patient's 
skull of an entry point which would allow access to a desired sulcus 
for surgery. Figure 10 illustrates the end result of the example. 

25 Surgeons now have the ability, with the aid of the present 

invention, to view important internal structures in relation to visible 
externa] structures and to mark the location of internal structures 
on the patient's skin. Surgeon also are now allowed to interactively 
view the internal structures in relation to visible external structures 

30 during the operation, even if the patient moves from his original 
position, thereby allowing more precise and accurate surgery. 

While several presently preferred embodiments of the novel- 



10/26/06, EAST Version: 2.1.0.14 



WO 94/24631 



-14- 



PCT/US94/02572 



turbulence imaging system have been described in detail herein, 
many modifications and variations will now become apparent to 
those skilled in the art. It is, therefore, to be understood that the 
appended claims are intended to cover all such modifications and 
5 variations as fall within the true spirit of the invention. 
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WHAT IS r?LATMF,n TR- 

1. A real-time surgery apparatus for displaying to a surgeon 
interactive internal and external images of a patient comprising: 

a) a medical imaging system for obtaining three-dimensional (3D) 
imaging data of internal structures of said patient; 
5 b) an imaging system for providing real-time images of exposed 
surfaces of said patient from a location (x,y,z) and orientation 
angle (a,$,9) with respect to the patient; 

c) a workstation for receiving the 3D imaging data and creating 

images of internal structures of said patient viewed from 
10 location (x,y,z) and orientation (a,<J> f 9); 

d) mixer means for creating a video signal being a desired mixture 
of the images of exposed surfaces and internal structures; and 

e) display means for displaying an image from the video signal to 
said surgeon to assist in surgery. 

2. The real-time surgery apparatus of claim 1 further comprising a 
tracking device adapted to measure the location (x,y»z) and 
orientation (a,<|>,6) of the imaging device with respect to said 
patient, for repeatedly providing these measurements to the 

5 workstation. 

3. The real-time surgery apparatus of claim 1 wherein the 
workstation further comprises surgical planning means adapted 
to create models of said patient's internal structures, interactively 
manipulate the models to result in modifications to internal 

5 structures, store the models and modifications and display the 
models and modifications at the proper orientation and scale to 
coincide with other images being displayed. 

4. The real-time surgery apparatus of claim 1 wherein the 
workstation further comprises a standard anatomy workstation 
adapted to create normal anatomical models of internal 
structures, and to display the models at the proper orientation 

5 and scale to coincide with other images being displayed, such that 
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said workstation can act as a guide in reconstructive surgery. 

5. Real-time surgery apparatus for displaying to a surgeon 

interactive three-dimensional (3D) internal and external images 
of a patient comprising: 

a) a medical imaging system for obtaining three-dimensional (3D) 
5 imaging data of internal structures of said patient; 

b) an imaging system for providing stereoscopic right and left real- 
time images of exposed surfaces of said patient viewed from a 
right and left location (xi,yi,zi) {x2,y2,Z2) and right and left 
orientation angle (ai,<j>i,8i) (azfafa)* respectively relative to the 

10 patient; 

c) tracking means for measuring locations (xi,yi,zi) (x2,y2,Z2) and 
orientation angle (ai4i,9i) faMfii) of the imaging device 
relative to said patient; 

* d) a first workstation for receiving right location (xi,yi,zi) and 
15 orientation angle (cti,4i,8i) from the tracking means, and for 

receiving 3D imaging data from the medical imaging system 
and creating a right computer generated image of internal 
structures of said patient as viewed from location (xi,yi,zi) and 
orientation angle (ai,$i,0i); 
20 e) a second workstation for receiving left location (*2,y2»Z2) ^ 
orientation angle (ct2,$2,62) from the tracking means, and for 
receiving 3D imaging data from the medical imaging system 
and creating a left computer generated image of internal 
structures of said patient as viewed from locations <X2,y2»z2) and 
25 orientation angle (a2,ta»92); 

f) mixer means for creating a plurality of right view images and a 

plurality of left view images each of the plurality of images being 
a desired mixture of the right and left real-time video images of 
exposed surfaces and right and left computer generated images 
30 of internal structures, respectively; 

g) stereoscopic display means for providing right view images to 
said surgeon's right eye, and left view images to said surgeon s 
left eye thereby simulating a 3D image of internal and external 
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structures of said patient 

6. A method of facilitating surgery comprising the steps of: 

a) acquiring 3D medical imaging data from internal structures of a 
patient; 

b) acquiring real-time images of exposed surface viewed from a 
5 location (x,y,z) and orientation angle (<x,<fr,e) with respect to a 

patient; 

c) creating computer generated images of the internal structures 
from the 3D medical imaging data viewed from a location (x,y,z) 
and orientation angle (a,4>,6); 

10 d) mixing portions of the real-time video images of the exposed 
surfaces with portions of the computer generated images to 
produce a mixed video signal; and 
e) displaying the mixed video signal to provide interactive images 
with both exposed structures and internal structures to assist 

15 an operator in surgery. 

7. The method of facilitating surgery of claim 6 wherein said real- 
time images are acquired from an imaging system, and further 
including the step of tracking the location (x,y,z) and orientations 
(ct,$,8) of the imaging device. 

8. The of claim 6 further comprising the step of creating a computer 
generated guide image of normal internal structures viewed from 
a location (x,y,z) and orientation angle (<x,$,8) to be used as a 
guide to reconstructive surgery; and mixing portions of the guide 

5 image with the images produced in step "d". 

9. A method of aiding a surgeon in surgery comprising the steps of: 

a) obtaining three-dimensional (3D) imaging data of internal 
structures of said patient; 

b) providing stereoscopic left and right real-time images of exposed 
5 surfaces of said patient viewed from a left and right locations 

(xi,yi»zi) (x2,y2»Z2) and left and right orientation angles (cti,<j>i,8t) 
(<*2>$2i02)> respectively relative to the patient; 
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c) measuring the locations (xi,yifZi) (x2,y2>Z2) a**d orientation 
angles (ai4i,8i) (ot2,<|>2,62)> respectively; 

d) creating a left computer generated image and a right computer 
generated image of internal structures from the 3D imaging 
data of said patient viewed from locations (xi,yi,zi) (x2,y2>Z2) and 
orientation angles (cti,<fri,9i) (a2,$2>02) respectively; 

e) mixing a desired portion of the real-time video image of exposed 
surfaces and computer generated image of internal structures 
viewed from location (xi,yi,zi) and orientation angle (ai,4>i»0i) 
for creating a left view video signal, and mixing a desired 
portion of the real* time video image of exposed surfaces and 
computer generated image of internal structures viewed from 
location (X2,y2>z2) and orientation angle (ctiMfa) for creating a 
right view video signal; and 

f) alternately providing images from the left view video signal to 
said surgeon's left eye, and from the right view video signal to 
said surgeon's right eye, thereby simulating a 3D image of 
internal and external structures of said patient to aid the 
surgeon in surgery. 

10. The method of aiding a surgeon in surgery of claim 9 further 
comprising the step of slightly adjusting the scaling, location and 
orientation of the left and right computer generated images to 
correspond to the left and right real-time images of exposed 

5 surfaces. 

11. The method of facilitating surgery of claim 6 further comprising 
the step of marking external surfaces of said patient 
corresponding to the position of underlying internal structures 
desired to be reached. 

12. The method of facilitating surgery of claim 11 further comprising 
the step of cutting on the marked exposed surface to reach the 
underlying desired internal structures. 
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